Repository navigation
[Perf] [GB300] qwen3_5: attention, GDN front and out_proj, shared expert and the MTP draft in the plugin; TP4, and two and four requests at TP8 behind an opt-in - #20
Merged
Conversation
4 of 5 tasks
YAMY1234
force-pushed
the
qwen35-gb300-full
branch
from
October 21, 2026 02:22
e94056d to
9abccc5
Compare
4 of 5 tasks
YAMY1234
force-pushed
the
qwen35-gb300-full
branch
from
October 21, 2026 17:12
24e4b51 to
33cf2e8
Compare
YAMY1234
marked this pull request as ready for review
October 21, 2026 17:26
…B MTP verify on GB300 TP8 The target-verify step of one request (7 rows) runs the plugin's stock step: SGLang's ops in SGLang's order over the weights and pools SGLang loaded, read through SGLang's EAGLE binding. The parity tool (SGLANG_NVIDIA_MEGAKERNEL_PARITY=1) compares it with SGLang's forward bit for bit and reports how far apart they are. The MTP draft spec covers no step yet, and a slot list naming verify_sample is refused with the switch that replaces it. Opt-in slots (SGLANG_NVIDIA_MEGAKERNEL_SLOTS), each bitwise: - route: the router GEMM and the NVFP4 MoE input in one launch; - mkm: on route's outputs, the top-10, FC1, SwiGLU, requantization and FC2 in one launch; - gdn_verify: the GatedDeltaNet verify chain with four launch choices changed; - gdn_pdl: its ReplaySSM verify launched as a programmatic dependent of the conv; - hop: the boundary's three fused all-reduce + residual + RMSNorm points on an NVLink multicast hop.
…the hop's push GEMM (numeric) The GatedDeltaNet's static-FP8 out_proj (1-8 rows) runs as one GEMM whose epilogue pushes the bf16 partial into the decode hop's mailbox, and the MoE input's all-reduce then only consumes it. The hop's device protocol moves into q35_hop.cuh so other kernels can push from their epilogues. Slot gdn_push (with gdn_verify and hop); numeric: it sums in another order than FlashInfer's BMM.
… and conv in one launch (numeric) One launch of 11 clusters runs the static-FP8 in_proj_qkvz and BF16 in_proj_ba projections, the conv update and an L2 prefetch of the verify's state; the ReplaySSM verify and the gated norm into out_proj's FP8 input follow as programmatic dependents. Slot gdn_front (with gdn_verify and gdn_pdl, one request of 7 rows); numeric: the projections sum in another order than SGLang's GEMMs.
…p in the router front's launch (numeric) The router front's launch also quantizes the rows to the shared expert's static FP8 and computes its gate/up, so the shared expert on the alternate stream waits for that launch only. Slot route_shared (with route, up to 8 rows); numeric: the gate/up sums in another order than the FP8 linear's GEMM.
…plit-KV attention (numeric) After the fused FP8 K / V write, the full attention's core runs SKV2, the split-KV attention of SGLang branch nvidia/qwen35-agentx-gb300 (a CuTe DSL producer over 148 page splits and a Triton merge); its KV-length graph variants keep SGLang's attention below 8192 tokens. Slot skv2; numeric: SKV2 splits the KV and sums in another order than trtllm-gen.
… and epilogue (numeric) MKA's prologue writes the FP8 query, the bf16 gate and the step's FP8 K / V rows into the pool, and its epilogue gates, quantizes and runs o_proj into the TP-partial output (the kernels of SGLang branch nvidia/qwen35-agentx-gb300, built with NVRTC). Slot mka; numeric: its qkv and o_proj GEMMs sum in another order than SGLang's.
…its rows into the hop Where the MoE input's all-reduce runs the hop, MKA's epilogue pushes its output fragments into the hop's mailbox and that all-reduce only consumes them. Slot mka_push (with mka and hop); the pushed values are mka's own, so it adds no tolerance.
…EAGLE entries on SGLang's own ops The draft spec offers draft_loop (with begin_draft_round) and draft_extend for one request when they are requested in SGLANG_NVIDIA_MEGAKERNEL_SLOTS. Each runs inside SGLang's draft graphs over SGLang's loaded draft modules with SGLang's own ops (bitwise), the base the draft slots replace.
…one launch (numeric) Inside draft_loop, a one-row draft step's MoE (router GEMV, softmax top-10, routed experts and gated shared expert) runs as one launch of one CTA per SM on the bf16 weights SGLang loaded. Draft slot draft_moe; numeric: the dot products sum in another fp32 order, so the draft's claim turns into TOLERANCE.
…t-KV decode (numeric) Inside draft_loop, a one-row draft step's decode attention runs SKV2-Q1: trtllm-gen's KV split and rounding points, the CTAs merged in fp32 by a small Triton kernel. The draft loop's graph variants keep trtllm-gen below 16384 tokens of KV. Draft slot draft_q1; numeric: only the fp32 association differs from trtllm-gen.
…y's fc split, exchange and head (numeric) Inside draft_loop, the greedy one-row draft step boundary runs in two launches instead of seven: the fc as a row split, the LM head as a GEMV on the vocabulary shard, and one multicast exchange for the argmax, the positions and the next step's fc. The hidden-half fc partial declares one CTA per SM, as it launches, so NVRTC's build issues its weight loads together. Draft slot draft_xchg; numeric: only the fp32 summation order differs from SGLang's fc and LM head.
…points and o_proj push on the decode hop (numeric) Inside draft_loop, the draft step's two all-reduce + residual + norm points run as hops on the target's decode hop, in the rounding of the kernels they replace, and o_proj runs as a one-row bf16 GEMV whose epilogue pushes into the FFN input's hop. Draft slot draft_hop (with hop); numeric: the GEMV sums in another fp32 order than SGLang's GEMM.
… front writes the FP8 KV in the same launch After the qkv projection, one Triton launch runs the q / k Gemma RMSNorm, the RoPE and the gate split, and also writes the FP8 query and the K / V rows into the FP8 KV cache, in place of SGLang's second launch. Draft slot draft_front; bitwise: the same bytes as SGLang's two launches.
…ention on the split-KV attention (numeric) Inside draft_extend, the draft extend's decode attention runs on slot skv2's split-KV attention over the draft's KV pool, with the extend's KV-length graph variants (SGLang's attention below 32768 tokens). Draft slot draft_skv2 (with draft_extend); numeric: SKV2 sums in another order than trtllm-gen.
…traight into SGLang's outputs The served draft extend gathers its selected rows into hidden_out, and the exchange writes the token and its 1.0 into topk_index_out / topk_p_out, so the graph's three closing copies go. Entry draft_extend with slot draft_xchg; bitwise: the values are the same.
…ghts by one bulk copy into shared memory Slot gdn_front: each projection CTA brings its 128 KB of packed FP8 in_proj_qkvz weights into shared memory with one TMA bulk copy issued before its dependency wait, instead of 32 register loads per thread. The MMA sequence is unchanged, so every output is the same (bitwise).
…tic dependents Slots skv2 and draft_skv2: the split-KV producer launches as a programmatic dependent of the attention's front and loads its leading K / V tiles before it waits, and the merge launches as the producer's dependent. Same arithmetic, same bytes (bitwise).
…rogrammatic dependents Slot draft_q1: the SKV2-Q1 producer launches as a programmatic dependent and loads its leading K / V tiles before it waits, and the merge waits before it reads the partials; slot draft_front's one-row front releases its dependents at entry. The arithmetic is unchanged (bitwise).
…entry, with a shared-memory pad Slots draft_xchg and draft_hop: the draft step's final all-reduce + norm hop releases the draft head at its entry, and the head launches with 120 KB of unused shared memory per CTA so that it starts only on SMs the draft MoE has left. Each CTA's rows and arithmetic are unchanged (bitwise).
…the side-stream fc partial Slot draft_xchg: on the ranks whose fc slice is in the hidden half, the next step's hidden-half fc partial launches on 24 CTAs and the draft head on the remaining SMs beside it. Each row is still one warp's in the same arithmetic, so the logits, the partial and the token are the same (bitwise).
…atic dependent of the qkv GEMM Slot draft_front: with one row, the front loads its static operands (the norm weights, the M-RoPE axis map, the cache scales) before its dependency wait and issues every other load right after it. The arithmetic is unchanged (bitwise).
…utput gate as it stages its input Slot draft_hop: the one-row o_proj GEMV takes x * sigmoid(gate), computed while it stages x in the instruction sequence of SGLang's gate kernel, so that kernel no longer runs, and the q1 merge releases the GEMV at entry. The arithmetic is unchanged (bitwise).
… for intermediate 256 per rank The spec takes TP sizes 4 and 8. Slot mkm picks its engine from the intermediate size a rank holds, adding the TP4 engine of SGLang branch nvidia/qwen35-agentx-gb300 (bitwise against trtllm-gen), and slot draft_hop keeps SGLang's o_proj at TP4. Every other slot builds as at TP8, except gdn_front, which refuses at TP4.
…s of 2 and 4 requests With multi_request in the slot list, TP8 also covers the verify steps of 2 and 4 requests (14 and 28 rows), each slot only up to the rows its kernels serve: route 16, hop 15, gdn_push, route_shared and mka 8, mkm 7, gdn_front and skv2 one request, gdn_verify and gdn_pdl every width. Off by default; TP4 keeps one request. The parity check counts the requests' own rows apart in graph steps. Each slot keeps its claim.
…16 rows in one launch The 16-token engine of SGLang branch nvidia/qwen35-agentx-gb300 (a CTA per (token, top-k) slot, no clusters) at the TP8 expert shapes runs slot mkm's steps of 8 to 16 rows. Slot mkm_wide (with mkm and multi_request; vacuous where no covered step is wider than 7 rows); bitwise against FlashInfer's trtllm-gen MoE.
…ith multi_request A draft step of two requests runs the split fc, the hop's all-reduce points and the two-token MoE; q1, the o_proj push and the exchange stay one-request. It serves them where multi_request covers the verify steps of several requests (TP8); TP4 keeps one request. Numeric as the draft slots.
YAMY1234
force-pushed
the
qwen35-gb300-full
branch
from
October 21, 2026 18:47
33cf2e8 to
c5c714d
Compare
# Conflicts: # .github/CODEOWNERS # README.md # sgl_nvidia_mega/plugin.py # test/unit/test_contract.py # test/unit/test_plugin.py
nvrohanv
added a commit
to nvrohanv/sgl-nvidia-mega
that referenced
this pull request
Oct 29, 2026
Main's SGLANG_MIRRORS check (sgl-project#20) digests a function's tokens in another encoding than this branch's copy, which the merge replaced with main's. Pin the 18 GLM-5.3 mirrors with main's mirror_digest of the same SGLang code: no mirrored body changes, only how its digest is written.
4 of 5 tasks
YAMY1234
added a commit
that referenced
this pull request
Nov 10, 2026
… NVFP4 MTP target verify and NextN draft (#24) * [Feat] [B300] glm5_3: NVIDIA megakernel specs for GLM-5.3 NVFP4 MTP target verify and NextN draft on B300 TP8 Add sgl_nvidia_mega/glm5_3: the GLM-5.3 target spec for SGLang's megakernel layer (MTP verify steps of 1 to 4 requests, 4 to 16 rows, on 8 x B300 at TP 8), its step functions with stock paths adapted from SGLang's model code, and the CuTe DSL attention, small-M NVFP4 MoE and all-reduce hop kernels. With --enable-nvidia-megakernel every target slot runs (attention, moe, hops, qkv_a_small_m_shared, dnorm); SGLANG_NVIDIA_MEGAKERNEL_SLOTS is a test and bisect override. Needs SGLang's DSA binding view. Add the GlmMoeDsaForCausalLMNextN draft spec (slot draft, on by default) and its CuTe DSL kernels: the embedding gather and eh norms in one kernel, the column-sharded eh projection gathered by the target's all-reduce hop, the distributed top-1, and the draft layer's MLA prologue, in every captured draft graph. Needs embedding replication (with the flag, SGLang's megakernel layer sets SGLANG_ENABLE_EMBED_REPLICATION when it is unset), greedy draft acceptance and SGLang's EAGLE draft graph capture check; the contract test skips the check's marker, naming it, on an SGLang without it. At startup rank 0 logs the covered verify widths (1-4 requests) and the derived weight copies' memory per GPU, and warns when --max-running-requests is set above 8, where the megakernel helps little. Also the README section, a NOTICE paragraph, a CODEOWNERS line and the contract test's checks of the DSA binding view reads and the mirrored SGLang code; the GLM-5.3 tests are filed separately. The same specs serve GLM-5.2 NVFP4, which has GLM-5.3's architecture (GlmMoeDsaForCausalLM) and the same value of every config field the target and draft specs pin. The spec names and the plugin's log, refusal and rank agreement texts name both models, and the module path stays glm5_3. test/unit/test_glm5_3_glm5_2_config.py runs the published config of nvidia/GLM-5.2-NVFP4 (revision 53e0691e21895a3863a606dfd12910c69eba94ab) through SGLang's requirement check for both specs, and checks that a config that differs in any pinned value, or lacks one, is still refused. The package imports SGLang's megakernel interface from its location since sgl-project/sglang#45389 (sglang.kernels.plugin_kernels.nvidia, sglang.srt.kernel_plugins.nvidia) and falls back to sglang.srt.megakernels on an older SGLang, as deepseek_v41 does. * docs: correct GLM GPU test availability * [Feat] [B300] glm5_3: serve MTP with 5 draft steps (6 verify rows a request) The GLM-5.3 / GLM-5.2 specs served MTP / EAGLE with 3 draft steps only, so the 5 / 1 / 6 setup of SGLang's GLM-5.3 and GLM-5.2 B300 NVFP4 low-latency recipes stopped at startup with --enable-nvidia-megakernel. The verify widths, the MoE row map and the draft eh rows are now derived from the verify rows a request T (speculative_num_draft_tokens = speculative_num_steps + 1, top-1 drafts), and the specs serve the tested values 4 and 6 (plan.SERVED_TOKENS_PER_REQUEST); any other value is refused at startup, naming them. - Widths: every whole number of requests up to the kernels' 16 rows (plan.supported_rows): 4 / 8 / 12 / 16 rows at T 4 as before, 6 / 12 rows (1-2 requests) at T 6. Steps of more requests run SGLang's path. - MoE: the engine takes at most 4 row groups of at most 4 rows, token sets over one expert list rather than requests. moe_split groups the rows by 4 with the remainder last: T 4 keeps (4,) to (4, 4, 4, 4); 6 rows run on the two-group engine as (4, 2) and 12 rows on the four-group engine as (4, 4, 4). The one-group engine holds 4 rows, so it cannot take 6. - Draft: the eh_proj shard serves the draft decode's 1-4 rows and the draft extend at the covered widths, so T 4 keeps its rows and T 6 adds 6; remove the unused max_requests. - Attention, the hops and the draft attention already take any row count up to 16 on their 8- and 16-row forms; no kernel changes. At T 4 every width, row map, eh row set and kernel form is unchanged. * [Perf] [B300] glm5_3: the one-request fast forms at 6 verify rows With 5 draft steps a one-request verify step has 6 rows, past the 4-row limit of the forms that serve one request of 3 draft steps. Serve them at up to 6 rows: - the split MoE fold hop (SPLIT_MAX_ROWS 6; still taken only where the cooperative occupancy holds 2 x rows + 1 clusters of 8 CTAs, probed at build); - the plain all-reduce hops on the dense layers (DENSE_MAX_ROWS 6); - the 16-warp sparse MLA (one CTA per SM; its kernel already takes any row count and gives the 8-warp form's bytes). No width of 4 verify rows a request has 5 or 6 rows, so those steps keep their forms. The kernels are unchanged. * docs: glm5_3 MoE groups and eh_proj rows with 6 draft tokens The README's GLM-5.3 slot table named the MoE engine forms by requests and listed the eh_proj rows of 4 draft tokens only. Name the forms by row groups (6 rows run as 4 + 2, 12 as 4 + 4 + 4) and add the eh_proj rows of 6 draft tokens (6 and 12). * [Fix] glm5_3: pin the SGLang mirrors with main's mirror digest Main's SGLANG_MIRRORS check (#20) digests a function's tokens in another encoding than this branch's copy, which the merge replaced with main's. Pin the 18 GLM-5.3 mirrors with main's mirror_digest of the same SGLang code: no mirrored body changes, only how its digest is written. * [Fix] glm5_3: re-pin four SGLang mirrors for current SGLang main Four mirrored SGLang bodies changed on SGLang main. Each pin is now a tuple of the digests before and after the change, so the check passes on either SGLang: - DeepseekMLAForwardMixin.forward_absorb_prepare (sgl-project/sglang#46359): every changed line is on the --dcp-replicate-q-proj path, which runs only with decode context parallelism; the specs refuse it, and the stock attention step does not copy those lines. - DeepseekSparseAttnBackend._forward_trtllm (sgl-project/sglang#46990): the KV store and read go through the pool's layer router; on the DSA pool, a plain pool, that is the same set_mla_kv_buffer and get_key_buffer call with the same arguments, which the stock step makes. - DeepseekSparseAttnBackend.forward_decode: sgl-project/sglang#46990 as above, and sgl-project/sglang#44322, a triton branch for ROCm with decode context parallelism. - LogitsProcessor.forward (sgl-project/sglang#43449): the new lines run only with --enable-return-aux-hidden-states, which turns speculative decoding off, so never with the NextN draft; the output's new last_hidden_states stays None. The copied steps stay as they are. The other 14 mirrored bodies are unchanged on SGLang main. * [Fix] glm5_3: adapt the specs to SGLang's megakernel API version 2 SGLang main's megakernel contract is version 2: a built object may also offer the EAGLE entries draft_loop, draft_extend and begin_draft_round and attention graph_variants, which SGLang reads from every built object. The GLM-5.3 / GLM-5.2 target and draft built objects offer none of them (None: SGLang's own path, as before), and the specs declare API version 2, as main's DeepSeek-V4.1 and Qwen3.5 specs do. * docs: glm5_3 boundary docstring says plain dense hops up to 6 rows DENSE_MAX_ROWS is 6 since the 5-draft-step change; the module docstring still said 4. * test: glm5_3 pins the forward_mla mirrors after sglang#45946 sgl-project/sglang#45946 adds SAIL-only fp8 conditions to forward_absorb_prepare / forward_absorb_core and a FlashMLA-backend-only llama_4_scaling step to forward_absorb_prepare; the DSA path the spec serves is unchanged. Pin the new digests next to the old ones. * [Feat] [Rubin] glm5_3: serve SM107, TP8 across two trays, and nodes without NVLink multicast Port the Rubin pieces of #19 (glm_dsa) to the GLM-5.3 / GLM-5.2 specs. The kernels are the same; only the admission changes. - spec, draft: admit sm_107 (Rubin / VR NVL72) next to sm_103. The draft takes the target's ISAs. - spec, draft: declare Feature.MULTI_NODE_TP, so TP8 can span the two 4-GPU trays of an NVL72 rack (GB300 NVL72, Rubin VR NVL72), as the Qwen3.5 and Kimi K3 specs do. - runtime: SGLANG_NVIDIA_MEGAKERNEL_GLM_ALLOW_NO_AR_FUSION=1 (off by default) lets the model check and the per-step coverage pass where SGLang's FlashInfer all-reduce + RMSNorm fusion does not apply (no NVLink multicast). The stock residual steps then run SGLang's unfused all-reduce, which is what its boundaries run there, and slot hops runs its own symmetric-memory all-reduce. Without the variable nothing changes. - README: the GPUs and the two-tray launch. test/unit/test_glm5_3_rubin.py covers the three changes on CPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Perf] [GB300] glm5_3: Run the NextN draft-extend MLP on the select_index rows only The draft extend runs the NextN layer on every verify row of each request, but the next draft step reads only the select_index row (one per request); the logits processor already prunes to those rows before the LM head. After the post-attention norm the draft step now keeps those rows of the MLP input and of the residual SGLang's prepare wrote, so the MoE, its all-reduce and the final norm run on B rows instead of B * tokens (6 -> 1 and 12 -> 2 rows at EAGLE 5/1/6). The attention still writes every row's KV and indexer key. The hidden states handed to the logits then hold only the selected rows, with a LogitsMetadata that does not select again. - Slot draft_rowselect (shipped, a draft slot, needs slot draft). - Gate (draft_step.select_index_rows): DRAFT_EXTEND_V2, no logprobs, hidden capture not FULL, no padded-row mask, a 1-D select_index with 0 < B < N rows; the NextN MLP is DeepseekV2MoE without an all-to-all. Host-only facts, fixed per captured shape. draft_attention.select_rows raises on any other stream layout. - draft_attention.decoder_layer_forward takes the rows to keep (ffn_rows) and, at a width the draft attention family does not serve, SGLang's own attention call. - Rank 0 logs once per captured draft graph its attention and MLP rows. Numerics: the selection itself is exact (the kept rows equal the full-row stream's rows bit for bit through SGLang's residual stream and final add + RMSNorm). The draft MoE and its router pick their kernels by the row count, so the MLP output on B rows is not bit-identical to the same rows of the N-row run (draft.rowselect.mlp.* in the draft NUMERICS). Draft-only: target verification decides every emitted token. Tests: test/unit/test_glm5_3_draft_rowselect.py (CPU; the logits hand-off needs CUDA), test/kernels/test_glm5_3_draft_rowselect.py (4 GPUs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Feat] glm5_3: TP4 profiles, verify steps composed per width from kernel units The spec serves TP8 at 4 or 6 tokens per request up to 16 rows. It now also accepts TP4 at 3 to 6 tokens per request up to 48 rows (one GB300 tray: the InferenceX AgentX TP4 aggregate and PD decode-role configurations), and serves those widths with kernel units composed per step; the TP8 path (plan.build_providers, the executor, the families) is unchanged. - profile.py: the facts a build is decided on (TP, heads per rank, tokens per request, widths, device, features) and the accepted table: the TP8 envelope (plan.py) plus TP4 at 3-6 tokens, 48 rows. Profile.units says which build runs. - registry.py: candidates claim steps of a group atomically, with coverage per (profile, rows, mode) and contract tokens; per width the first preferred, switched-on, covering unit takes each step, the rest run the stock body. A decline (coverage) runs stock and is logged once per width; a family that covers a width but fails to build raises on every rank. A mixed group is a subclass of the stock group with the claimed steps bound to their units, so the stock step bodies and Providers.describe see them. - units.py: the TP4 preference table. Every unit is a slot (runtime.LL_SLOTS, shipped; OPT_IN_SLOTS only when listed), selected by SGLANG_NVIDIA_MEGAKERNEL_SLOTS as the TP8 families are; "" is parity (every accepted width on the stock bodies). - runtime/composed.py: the executor (GlmMegakernelVerify: its layer loop, first-capture checks and parity switch) on the composed step groups; its logits take ll.gemm's lm_head role and share the model processor's gatherer. runtime.build hands a TP4 profile to it. - spec.py: tp_sizes {4, 8}, the (TP, tokens) pairing checked against the profile table (profile_reasons), max_rows 48, the units' named tolerances. - ll/__init__.py, ll/jit.py: the units' package (optional modules, the ll.linear GEMM entry point, named tolerances) and the JIT loader for its .cuh kernels (SGLang's load_jit; refuses to compile under capture; failures raise). Tests: test/unit/test_glm5_3_{profile,registry,spec,ll_jit,verify_logits}.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Perf] [GB300] glm5_3: TP4 small-M BF16 GEMMs on PDL split-K and TGV (ll.gemm) The TP4 roles' small-M BF16 GEMMs that SGLang leaves on cuBLAS launch without programmatic dependent launch. ll.gemm moves them, at the row counts where a GB300 crossover bench measured a win in the GEMM's model context, to a PDL kernel (slot ll.gemm.<role> each): - router (FP32 logits): split-K at 17-96 rows (the tiny GEMM keeps <= 16); - indexer wk|weights_proj: split-K at 1-64 rows; - o_proj TP4: TGV 1-72; dense_down TP4: TGV 1-48; NextN eh_proj: TGV; - lm_head TP4: TGV 1-60; measured but deferred (run only when listed): the TP8 o_proj / q_b shapes. gemm_plan.py holds the plan and the named tolerances. The plan applies only on SM103 with 152 SMs; elsewhere, and for quantized or biased weights or unplanned rows, every role runs SGLang's own GEMM for the module, bit for bit. Split-K is a vendored copy of FlashInfer v0.7.2's cluster split-K kernel with FP32 output / any M and synchronization fixes (owner wait at .acquire.cluster, phase-0 wait before the TMEM release, tcgen05 thread-sync fences; no arithmetic change). FlashInfer's own kernel is used only when it carries those fixes (SplitKTactic.SYNC_FIXED); SGLANG_NVIDIA_MEGAKERNEL_GLM_GEMM_SPLITK pins auto / flashinfer / vendored. TGV is SGLang's cutedsl_bf16_gemm. Tests: test/unit/test_glm5_3_ll_gemm_plan.py (CPU: tactics, engagement only on SM103 / 152 SMs, slots, FlashInfer detection, stock dispatch); test/kernels/test_glm5_3_ll_gemm.py (1 GPU: path per role and row, tolerances vs cuBLAS and FP64, graph replay, PDL edges, launch stress). benchmark/bench_glm5_3_ll_gemm.py re-checks the crossover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Perf] [GB300] glm5_3: TP4 sparse MLA flash decode as unit ll.sparse_mla_fd The sparse_mla and uv_proj steps of the TP4 verify attention and of the NextN draft layer's attention on a flash-decoding CUDA kernel over the top-k gathered FP8 KV rows, instead of TRTLLM-Gen's sparse MLA decode + torch.bmm. The step reads the running forward's ForwardView (view.dsa, view.kv_pool, view.metadata), as the stock step does. - Kernels (ll/csrc/attention/glm_sparse_mla_fd_*.cuh, adapted from TensorRT-LLM's sparse MLA flash decode): one 16-CTA cluster per row, 8 or 16 heads; a v_b fold instance (8 heads, bitwise cuBLAS's BMM); a multi-tile mode (clusters of 2-8 CTAs, 8 heads). - Launch plan (GB300, 152 SMs): one-tile up to 7 rows; built at compile; the co-resident cluster counts are agreed over the TP group (MIN); a build failure on any rank raises on every rank. - Covers sm_103 with 152 SMs, 16 heads (TP4), <= 7 rows; declines with TRTLLM-Gen skip-softmax decode on. - Tolerance ll.sparse_mla_fd.max_abs = 0.0625 against TRTLLM-Gen (measured 0.052 / 0.055 at 8 / 16 heads). 16 heads: -2.1 to -2.8 us per layer at 5 / 6 rows (benchmark/bench_glm5_3_sparse_mla.py). Tests: numerics vs TRTLLM-Gen and an FP64 oracle, the provider steps vs the stock steps, CUDA-graph replay and the PDL chain (1 GPU); the launch plan and rank agreement (CPU). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Perf] [GB300] glm5_3: TP4 attention front on low-latency fused-A and a one-launch MLA KV half The unit ll.attention (groups attention and draft_attention) serves the attention front of a TP4 rank (16 heads, SM103, 1-48 rows, every step mode) by claiming qkv_a / prologue / index_k / rope_cache of AttentionStep: - qkv_a: SGLang's min-latency fused-A kernel with a larger stage ring and a pipelined MMA loop from 9 rows, up to 16 rows (bit for bit SGLang's kernel); a split-K form at 17-24 rows when one wave fits (ll.attention.qkv_a_wide.max_rel of fp32); stock above 24 rows. - prologue: stock q_a norm; q_b on the pipelined form at 9-16 rows (bitwise); the kv_a norm deferred into rope_cache when folded. - index_k: the stock body with its key / head-gate projection through ll.linear("wk_weights_proj"). - rope_cache: one PDL launch for the kv_a norm (folded), RoPE and FP8 of q and k and the MLA cache append, bit for bit FlashInfer's norm and mla_rope_quantize_fp8 and SGLang's set_mla_kv_buffer (the cache through the step's ForwardView). The fold (slot ll.attention.kv_norm_fold) engages only when FlashInfer's norm backend, read from its public surface, is CuTe DSL and a probe at build gives the stock kv_a norm's bits; the family logs which way it went. fused_a_ll.cuh is adapted from SGLang's dsv3_fused_a_gemm.cuh. Tests (1 GPU): fused-A forms bitwise at 1-16 rows and within tolerance at 17-24, graph replay; the KV half bitwise (q and every cache byte); the stock AttentionStep vs ll.attention on one TP4 layer at 5-48 rows, eager and replayed. CPU: the norm-backend decision, coverage and composition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Perf] [GB300] glm5_3: TP4 all-reduce + residual + RMSNorm hops on a plugin-owned Lamport buffer The unit ll.hops serves the three hop steps (HopSteps) of the TP4 verify step: - csrc/distributed/glm_ll_mnnvl_oneshot.cuh: one-shot / two-shot MNNVL all-reduce + residual + RMSNorm with a MoE-finalize prologue and an NVFP4 epilogue (builds with nvcc 13.0 and 13.4). Every wait on a peer is bounded: past the deadline it writes a host-mapped error record and traps (fail-stop). - ll/transport.py: the Lamport buffer and flags are the plugin's own (torch symmetric memory, or FlashInfer's McastGPUBuffer) in FlashInfer's MNNVL layout; no FlashInfer workspace or flags are shared. Modules load through ll.jit; a build failure raises on every rank. - ll/hops.py: the GB300 launch plan per world size and row count (W=4: one-shot up to 36 rows, two-shot 37-48), the family (deadline min(20 s, the build context's), o_proj through ll.linear) and its step group. Covers the TP4 verify step on sm_103 up to 48 rows; provides hops:finalize and mlp_input:nvfp4 (written only when a taken MLP unit consumes it). Tests: the plan tables (CPU); at W=4 every row 1-48 bit-identical to FlashInfer's one-shot / two-shot, the finalize hop bit-identical to moe_finalize_fuse_shared + the hop, a protocol mix eager and in a CUDA graph; fail-stop of every cross-GPU wait; a two-tray multicast probe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Perf] [GB300] glm5_3: TP4 MoE and dense MLP steps with the deferred finalize (ll.moe) The units ll.moe.nvfp4 and ll.moe serve the MLP step (MlpSteps) of the TP4 verify step and leave the MoE finalize to the next hop: - MoE layers: the router GEMM (ll.linear("router"), FP32 logits), then FlashInfer's public trtllm_fp4_block_scale_moe with do_finalize=False and the layer's processed NVFP4 weights (SGLang's fused_experts_none_to_flashinfer_trtllm_fp4 call for this configuration), and the shared expert on the layer's alternate stream; returns a deferred finalize for the next hop. - ll.moe.nvfp4 takes the routed experts' FC1 input from the post-attention hop's NVFP4 epilogue; ll.moe quantizes its own input. - Dense layers: the stock gate_up_proj and SiLU-and-mul, then ll.linear("dense_down"), un-reduced. - Opt-in slot ll.moe.alt_stream_priority: the shared expert on a stream of the greatest priority at <= 6 rows. It gains only with FlashInfer's small-batch routing kernel (TP4 5/1/6: -1.82% step at 6 rows); on stock FlashInfer 0.7.2 it loses (+1.2% at 5 rows, TP4 4/1/5, n = 2; +2.4% at 6 rows), so it runs only when listed in SGLANG_NVIDIA_MEGAKERNEL_SLOTS. - Both require hops:finalize. A layer that does not fit declines the family with the layer and the reason. Tests (4 GPUs): the copied call equals SGLang's runner bit for bit at 5-48 rows; the layer (hop + MoE + finalize hop) equals the stock layer bit for bit on the hop's FP4 input inside FlashInfer's one-shot band; CUDA-graph replays equal eager; the priority stream keeps the bits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Perf] [GB300] glm5_3: TP4 NextN draft composed from units (P6a eh_proj, LL attention, row select) The draft step (draft_step.FusedDraft) serves TP8. At a TP4 profile the draft spec now builds draft_units.ComposedBuiltDraft: each draft graph (draft decode at one row per request, draft extend at the verify rows) is composed through the registry from the draft step groups (draft_front, draft_eh, draft_attention, draft_mlp, draft_top1): - draft.front (slot draft): the draft step's fused embedding + eh norms (the kernel does not depend on the TP size); - ll.draft_eh (P6a, ll/draft.py): eh_proj row-parallel, each rank's K window times its weight columns. With the target's LL hops (found through the shared-table registry, runtime.register_target_hops) the partials are summed and the draft layer's input norm applied in one hop launch, and the layer starts from that (draft_attention.PreparedInput: the attention prepare is skipped); otherwise SGLang's all-reduce and the stock norm. Per draft forward at TP4, L2 flushed: 20-26 us vs 39-41 us for the replicated GEMM; - the draft layer's attention on ll.attention and ll.sparse_mla_fd (the stock DraftAttention on the step's ForwardView elsewhere); - draft_rowselect in the draft extend (draft_step.select_index_rows); - the LM head on ll.gemm's lm_head role; the top-1 stays SGLang's (at TP4 its one-tray multimem gather is as fast as the distributed argmax or faster). The composed draft is built when the target shares its embedding and LM head (set_embed_and_head, before the KV pool is sized), so the LM-head role is built on the target's head. Draft graphs outside the profile's widths run SGLang's draft forward. The TP8 draft keeps the TP8 widths (<= 16 rows). Tests: test/unit/test_glm5_3_draft_units.py and test_glm5_3_draft_eh_prepared.py (CPU: widths, compositions, slots, the hand-off to the logits processor), test/kernels/test_glm5_3_draft_eh_proj.py (4 GPUs: numerics at 1-48 rows, the hop sum + norm, graph replay). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Test] glm5_3: Serving tests and decode A/B configuration at the AgentX TP4 / TP8 points - test/e2e/glm5_3_server.py, test_glm5_3_server.py: servers at the InferenceX AgentX configurations (tp4-416: aggregated TP4, EAGLE 4/1/5, HiCache; tp8-516: aggregated TP8 on two trays, EAGLE 5/1/6, HiCache; tp4-516: the disaggregated decode role, TP4, EAGLE 5/1/6, standalone): on / off (the active line, covered decode graphs, GSM8K-200 within 3 points), parity (SGLANG_NVIDIA_MEGAKERNEL_SLOTS="" vs flag off, greedy ids and logprobs bitwise) and HiCache load-back (load-back output equals the device-hit output). - benchmark/README.md: the GLM decode A/B protocol at those configurations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Feat] [GB300] glm5_3: Serve the TP4 decode role of a PD-disaggregated deployment A PD deployment of GLM-5.2 on GB300 (the InferenceX AgentX 1p5d-c30 / 1p3d-c40 points) runs a TP1 PP4 prefill role and TP4 EAGLE 5/1/6 decode roles. The spec now lists Feature.PD_DISAGGREGATION, so a decode role serves its verify steps on the TP4 kernel units: the KV arrives through the transfer backend (NIXL) into pool slots before a request's first verify step, outside the covered step, and the decode radix cache is not a framework feature. The TP8 envelope keeps PD refused (pd_reasons). The prefill role runs without the flag (TP1 and PP are refused anyway). Tests: test/unit/test_glm5_3_spec.py (the decode role accepted at TP4, refused at TP8); test/e2e/test_glm5_3_server.py TestGlm53PDDecode (GSM8K through a running 1P1D router, decode engaged, prefill not, no NIXL transfer failure). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Test] glm5_3: Remove the tests this PR added (repo policy: no tests in sgl-nvidia-mega) Per review, sgl-nvidia-mega carries no tests. Remove the 28 test files this PR added under test/e2e, test/kernels and test/unit (the GLM-5.3 TP4, draft row select and PD decode tests and their helpers glm5_3_server.py and glm5_3_slots.py), and drop the benchmark README's pointer to the removed test/e2e/glm5_3_server.py. Tests that already exist on the base branch are left untouched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * glm5_3: Keep SGLang's copyright notice in the headers of files adapted from SGLang Apache-2.0 requires derivative files to retain the copyright notices of the work they adapt. Following the OSRB guidance, add "Portions Copyright 2023-2024 SGLang Team (Apache-2.0)" after the NVIDIA SPDX lines of the glm5_3 files adapted from SGLang, and name the origin of ll/moe.py, whose routed-expert call copies SGLang's. fused_a_ll.cuh copies most of SGLang's dsv3_fused_a_gemm.cuh, so it keeps that file's header unchanged and adds the NVIDIA and SGLang lines below it. Header lines only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Fix] glm5_3: pin the SGLang mirrors for main's SGLang (cf6133a) main's unit tests check out SGLang cf6133a, where eight glm5_3 mirrors no longer matched. Each change keeps the covered steps as mirrored, so the new digest is added beside the old ones (both SGLangs pass): - forward_absorb_prepare / forward_absorb_core (#44503, #38943): the indexer, the unfused RoPE and the W_UV projection moved into the run_mla_indexer, prepare_mla_qkv and project_mla_values hooks, whose defaults are the old calls (no SGLang class overrides them); the DCP extend gather is out of the spec's setups. - Indexer.forward_cuda, DeepseekSparseAttnBackend._forward_trtllm (#46890): prefill-CP only. - DeepseekV2DecoderLayer.forward (#46857), in the hops, the MoE step and the draft attention: an assert under CP decode attention TP, which the spec refuses. - LogitsProcessor.forward (#47200): vocab-sharded logits for a target verify only; the draft steps the mirror covers take the gather as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Doc] glm5_3: Document the TP4 split-K GEMM source switch in the README SGLANG_NVIDIA_MEGAKERNEL_GLM_GEMM_SPLITK picks the split-K source of the TP4 small-M GEMMs (auto, flashinfer or vendored); document it next to the TP4 units, as the README documents Qwen3.5's SGLANG_NVIDIA_MEGAKERNEL_MOE_TACTICS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Yangmin Li <yangminl@nvidia.com> Co-authored-by: shuw <shuw@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Kaixi Matteo Chen <kaiximatteoc@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
This PR adds the Qwen3.5-397B-A17B (NVFP4) MTP megakernel for GB300: the target's verify step and the MTP draft at TP8 across two trays, and TP4 on one tray.
On the SGLang side it needs sgl-project/sglang#45266 (the Qwen3.5 binding, the EAGLE draft entries and the KV-length graph variants). sgl-project/sglang#45474 holds the runtime changes that turn the plugin's GPU savings into speed. The two SGLang PRs are independent; either can merge first.
With both SGLang PRs and this plugin, static decode at 128K context goes from 706.4 / 700.3 to 1291.7 / 1276.0 TPS/User (P50 / P90). That is 101.0% of the development build these kernels were taken from at P50. On InferenceX AgentX (TP8, concurrency 11) the formal score is X 1234.57, against the development build's 1219.51.
Modifications
One commit per optimization. Every slot is opt-in by name in
SGLANG_NVIDIA_MEGAKERNEL_SLOTS. "Numeric" slots change the order of floating-point operations; the others are bitwise equal to SGLang's ops.Foundation: the plugin's Qwen3.5 target spec, a stock step that runs SGLang's ops through the binding, and a parity tool (
SGLANG_NVIDIA_MEGAKERNEL_PARITY=1). It adds the first five slots, all bitwise:route: router GEMM and MoE input quantization in one launch;mkm: top-k, FC1, SwiGLU, requantization and FC2 in one launch;gdn_verify: the GatedDeltaNet verify chain;gdn_pdl: its ReplaySSM step as a programmatic dependent;hop: the layer boundaries' all-reduce + residual + RMSNorm on an NVLink multicast hop.Target slots, one commit each:
gdn_push: the GDN out_proj as the hop's push GEMM;gdn_front: GDN projections and conv in one launch;route_shared: the shared expert's gate/up in the router's launch;skv2: full attention on split-KV;mka: the fused attention prologue and epilogue;mka_push: the attention epilogue pushes into the hop.Draft entries
draft_loop/draft_extendon SGLang's ops, then one commit per draft slot:draft_moe;draft_q1: split-KV decode with KV-length graph variants;draft_xchg: fc split, exchange and head;draft_hop: all-reduce points and o_proj push;draft_front: front and FP8 KV write in one launch;draft_skv2.A further commit has the draft extend write its token and rows straight into SGLang's outputs.
Refinements, each its own measured optimization:
TP4 on one tray:
mkm's engine for intermediate size 256 per rank; the draft hop uses SGLang's o_proj at TP4.gdn_frontis TP8-only.multi_request(opt-in, TP8): slots also cover verify steps of 2 and 4 requests (14 and 28 rows). It is off by default, because it costs batch 1 about 0.3%.mkm_wide: routed experts of 8–16 rows on a 16-token engine. Requiresmulti_request.draft_loopserves two requests undermulti_request.Usage
# TP8 on two GB300 trays, SGLang with sgl-project/sglang#45266 (+ #45474 for its switches) SGLANG_ENABLE_EAGLE_POST_VERIFY_CUDA_GRAPH=1 SGLANG_ENABLE_GDN_PREFILL_GRAPH_EXTEND=1 \ SGLANG_NVIDIA_MEGAKERNEL_SLOTS=route,mkm,gdn_verify,gdn_pdl,hop,draft_loop,draft_extend,draft_moe,draft_q1,draft_xchg,draft_hop,draft_front,draft_skv2,gdn_push,gdn_front,route_shared,skv2,mka,mka_push \ python -m sglang.launch_server --model-path nvidia/Qwen3.5-397B-A17B-NVFP4-V2 \ --tp 8 --nnodes 2 --speculative-algorithm NEXTN --speculative-num-steps 6 \ --speculative-eagle-topk 1 --speculative-num-draft-tokens 7 \ --attention-backend trtllm_mha --enable-nvidia-megakernel ...For more than one request at TP8, add
multi_request,mkm_wide. At TP4, dropgdn_front.Accuracy Tests
multi_requeston (vs off)Speed Tests and Profiling
GB300, static decode with forced acceptance 4.80, about 1,780 rounds per server start. TPS/User = 4.80 tokens / round time. "Full stack" is main + sgl-project/sglang#45474 + #45266 + this PR. TP8 rows are the mean of two pairs of trays run in opposite orders.
multi_requeston, TP8 at 128K:draft_loop(101.4% of the development build);The head also runs on SGLang main after sgl-project/sglang#45112: it reads the renamed shared-expert count and accepts the moved ReplaySSM files. The speed and accuracy runs used SGLang main
28448a86d4and this PR's code at3509187d74, an earlier revision. The head adds themulti_requestopt-in (off by default, batch-1 tokens unchanged) and the two-requestdraft_loop, both measured above, and is otherwise the same code.